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(54) A network communication device including bonded ports for increased bandwidth 



(57) A network communication device including port 
control circuitry for controlling packet flow between the 
ports of the device, where the port control circuitry in- 
cludes a port manager that directs packets between the 
ports and port bonding circuitry that bonds two or more 
olthe ports into a bonded port set. For each packet to 
be sent via the bonded port set, the port bonding circuitry 
selects one of the bonded ports tor transmitting the 
packet. More than one bonded port set may be defined 
in a given communication device, and each bonded port 
. set may include from two ports up to all the ports of the 
device, as long as each port is included in only one 
bonded port set. One or more port bonding registers are 
provided to identity which ofthe plurality ofports are 
bonded in each bonded port set. In one embodiment, 
the bonded ports are selected on a packet by packet 
basis so as to achieve a relatively even distribution of 
packets sent by each bonded port. In an alternative em- 
bodiment bonded ports are assigned to packet source 
identifiers so as to achieve a relatively even distribution 
of source identifiers among the bonded ports. If bonded 
ports are assigned to particular source identifiers, then 
the traffic is preferably monitored and the assignments 
are periodically adjusted to achieve even distribution of 
packet flow on the bonded link. The bonded ports may 
have different bandwidths, in which case traffic is dis- 
tributed on a proportionate basis. 
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Description 

[0001] The present invention relates generally to net- 
works, and more specifically, to a network communica- 
tion device with port bonding architecture that enables 
multiple ports to be bonded together to provide a higher 
bandwidth connection between any two devices in a net- 
work system. 

[0002] Network communication devices are available 
in a variety of configurations and throughputs to provide 
connectivity between multiple network devices such as 
computers, servers, DTEs etc. A switch, for example, 
provides a significant advantage over other network 
communication devices, such as repeaters or hubs, by 
learning media access control (MAC) addresses of net- 
work devices and reducing extraneous packet transmis- 
sions. MAC addresses are assigned to network devices 
and are unique in the industry to distinguish one device 
from the next. Multiple switches may be included to ex- 
pand the number of network devices in a given network 
system. However, redundant connection paths between 
any two network devices or between two or more net- 
work communication devices result in packet looping. 
Packet looping is undesirable because it causes the 
generation of duplicate packets and/or broadcast 
storms. Significant packet looping eventually results in 
an excessive number of dropped packets and drastically 
reduced performance of a network. 
[0003] The spanning tree algorithm.was invented to 
locate all connection paths between resources on the 
network and disable redundant connection paths to 
avoid packet looping problems. Although a redundant 
link may be included between any two devices, the re- 
dundant link is typically only used to serve as a backup 
link if the primary link fails or otherwise becomes unus- 
able. 

[0004] While network communication devices ena- 
bled connection of multiple computers to one or more 
servers, the spanning tree algorithm allowed only one 
valid path at a time between any computer and a server 
If multiple computers coupled to one server via a com- 
mon link between two network communication devices 
attempted to access that server at the same time, the 
traffic demand of that link often became greater than its 
available bandwidth, thereby overwhelming the link. 
This usually resulted in dropped packets, disrupted con- 
nections and repeated packet transmission attempts. 
The common link therefore was a bottleneck between 
the two network communication devices. 
[0005] It is desired to increase the available band- 
width between any two network communication devices 
without substantial modification and without increasing 
the bandwidth of any given port. 
[0006] A network communication device according to 
the present invention includes a plurality of ports that 
are each used to receive and transmit packets of data 
and port control circuitry for controlling packet flow be- 
tween the ports. The port control circuitry includes a port 



manager that directs packets between the ports and port 
bonding circuitry that bonds two or more of the ports into 
a bonded port set. For each packet to be sent via the 
bonded port set, the port bonding circuitry selects one 
5 of the bonded ports for transmitting the packet. More 
than one bonded port set may be defined in a given com- 
munication device, and each bonded port set may in- 
clude from two ports up to all the ports of the communi- 
cation devices, as long as each port is included in only 

10 one bonded port set. One or more port bonding registers 
are preferably provided to identify which of the plurality 
of ports in the network communication device are bond- 
ed in each bonded port set. The present invention is ap- 
plicable to and is illustrated using network switches, al- 

'5 though the present invention is applicable to other net- 
work communication devices as well. 
[0007] In one embodiment, the bonded ports are se- 
lected on a packet by packet basis so as to achieve a 
relatively even distribution of packets sent by each 

20 bonded port in the bonded port set. A memory is pref- 
erably included in which the port control circuitry stores 
a plurality of entries, where each entry includes a source 
identifier provided in each packet and a corresponding 
port identifier. The source identifier is typically a MAC 

25 address of a sending device to uniquely identify that de- 
vice. If bonded ports are selected for transmission on a 
packet by packet basis, then for packets received at any 
of the bonded ports of the bonded port set, the port 
bonding circuitry programs the corresponding port iden- 

30 tifier to represent the bonded port set in general. In this 
manner, when a packet is received by the communica- 
tion device to be sent out the bonded port set, such as 
a un least packet with a destination identifier that is 
equivalent to a source identifier received at a bonded 

35 port, the communication device dynamically selects any 
one of the bonded ports in order to achieve an even dis- 
tribution of traffic on the bonded port set. Such dynamic 
selection on a packet by packet basis ensures even traf- 
fic distribution but may result in overhead associated 

40 with each packet. 

[0008] In an alternative embodiment, for each entry in 
the memory that includes a source identifier that was 
received at any bonded port of the bonded port set, the 
port bonding circuitry programs the port identifier to 

45 identify one bonded port of the bonded port set so as to 
achieve a relatively even distribution of source identifi- 
ers among the bonded ports of the bonded port set. 
Thus, each source identifier received at the bonded port 
set is assigned toa particular bonded port of the bonded 

50 port set. The selected bonded port is not necessarily the 
same port at which the packet was received. Instead, 
selection is made to achieve relative distribution of iden- 
tifiers among the bonded ports. In this manner, every 
unicast packet received by the communication device 

55 specifying a destination identifier equivalent to the 
source identifier is sent via the same selected bonded 
port of the bonded port set. 

[0009] Assignment of source identifiers to particular 
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bonded ports and approximately even distribution of 
source identifiers annong bonded ports is based on the 
assumption thai each device generates approximately 
the same amount of traffic overtime. In operation, a traf- 
fic imbalance may result so that one bonded port is car- 
rying a disproportionate amount of traffic. In an optional 
embodiment, the port bonding circuifry monitors unicast 
packets sent via each bonded port of the banded port 
set and periodically reprograms the port identifier of 
each of the plurality of entries in the memory so as to 
achieve a relatively even distribution of unicast packet 
traffic among the bonded ports of the bonded port set. 
In this manner, the bonded port assignments are dy- 
namically adjusted. 

[0010] In another embodiment according to the 
present invention including VLAN capabilities, each of 
the entries in the memory that has a source identifier 
that was received at any non-bonded port includes a bit 
map having a bit corresponding to each of the bonded 
ports. The bit maps are particularly useful for broadcast 
and multicast packet handling and to distribute broad- 
cast and multicast packets on bonded port sets. The port 
bonding circuitry programs each bit map of each of the 
plurality of entries to assign one bonded port of the 
bonded port set so as to achieve a relatively even dis- 
tribution of bonded port assignments to source identifi- 
ers corresponding to non-bonded ports. Additionally, the 
memory may include a default bit map table that in- 
cludes a default bit map for each of the plurality of ports 
for source f^AC addresses that have not been learned. 
In this case, each of the default bit maps include a bit 
corresponding to each of the bonded ports and the port 
bonding circuitry programs each of the default bit maps 
to assign one bonded port of the bonded port set so as 
to achieve a relatively even distribution of bonded port 
assignments among the plurality of ports. 
[0011] In the case of broadcast packets that are re- 
ceived from non-bonded ports and send to bonded ports 
as determined by the port control circuitry, the port bond- 
ing circuitry of the communication device monitors those 
packets sent by each bonded port of the bonded port 
set and periodically reprograms each bit map of each of 
the plurality of entnes so as to achieve a relatively even 
distribution of Ijroadcast packet traffic anrjong the bond- 
ed ports of the bonded port set. 
[0012] A packet-based network system according to 
the present invention includes at least two network com- 
munication devices, each including a bonded port set 
including at least two bonded ports. A bonded link is cou- 
pled between the bonded port sets of the respective 
communication devices, where the bonded link includes 
a connection link between respective bonded ports of 
the two devices. For each packet received by either de- 
vice to be sent to the other via the bonded link, each 
device selects one of its bonded ports. Such selection 
is made to evenly distribute packets on the bonded port 
link assuming each of the bonded ports operation at the 
same speed or a the same bandwidth. As before, the 



selection of bonded ports may be made on a packet by 
packet basis or by assigning a bonded port to each 
source identifier If bonded ports are assigned, a com- 
munication device evenly assigns source identifiers re- 
s ceived by its bonded ports among the bonded ports and 
evenly assigns bonded ports among the source identi- 
fiers received by its non-bonded ports. Such assignment 
may be random, round-robin, sequential, etc. 
[0013] The bonded port sets between two communi- 
10 cation devices may include bonded ports operating at 
different speeds or different bandwidlhs. If so, the de- 
vices coupled together via a bonded link each attempt 
to distribute traffic on the bonded link in proportion to the 
bandwidths of the bonded ports. Also, a network system 
IS according to the present invention may include multiple 
devices and multiple bonded port sets between respec- 
tive devices. Thus, a device may have multiple bonded 
port sets, each having any number of bonded ports, as 
long as only one bonded link is established between any 
two devices. 

[0014] A method according to the present invention 
includes steps of defining at least two ports as bonded 
ports forming a bonded port set, receiving a packet, de- 
termining that the received packet is to be sent via the 
bonded port set, selecting one of the bonded ports, and 
sending the packet via the selected bonded port. If a 
spanning tree function is applicable, a further step of 
modifying the spanning tree function to prevent disa- 
bling any of the bonded ports may be performed. 
[0015] It is appreciated that a network communication 
device with bonded ports according to the present in- 
vention enables a higher bandwidth capability between 
any two devices as compared to a single connection. 
Any number of bonded ports nnay be included in a bond- 
ed link between two communication devices to increase 
the available bandwidth by a proportional amount. Each 
device treats its bonded port set as though it were a sin- 
gle port. Thus, to avoid packet looping, each packet is 
sent via only one of the bonded ports and packets re- 
ceived at one bonded port are not re-transmitted on an- 
other bonded port of the same bonded port set. To 
achieve the highest bandwidth possible on the bonded 
link, each device distributes traffic on each bonded port 
as evenly as possible. The traffic distribution depends 
on the selection criterion for selecting bonded ports, 
where any one of several methods may be used, such 
as random, round- rob in, etc. The selection of a bonded 
port to send a packet is made either on a packet by pack- 
et basis or by source identifier (MAC address) assign- 
ment as previously described. In the assignment em- 
bodiment, the traffic may be dynamically monitored and 
the assignments periodically adjusted to achieve the 
highest bandwidth on each bonded link. 
[001 6] A better understanding of the present invention 
can be obtained when the following detailed description 
of the preferred embodiment is considered in conjunc- 
tion with the following drawings, in which; 
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Figure 1 is a simplified block diagram of a network 
system including bonded ports between two net- 
work switches implemented according to the 
present invention; 

Figure 2 is a perspective diagram that shows a net- 
work switch participating in mulTiple" bonded ports; 

Figure 3 is a diagram showing a bonded port set 
between two network switches where the bonded 
port set includes ports of different bandwidths; 

Figures 4A-4B are flowchart diagrams illustrating 
exemplary packet transmission operations of the 
network switches of Figure 1 ; 

Figure 5 is a diagram showing an exemplary broad- 
cast packet being transmitted through a bonded set 
of ports between two network switches without any 
duplication of the packet; 

Figure 6 is an exemplary block diagram of a network 
switch according to the present invention capable 
of supporting bonded ports; 

Figures 7A-7C are block diagrams illustrating the 
organization of the memory of Figure 6; 

Figure 8 is a flowchart diagram illustrating an exem- 
plary initialization and programming operation of 
the network switch of Figure 6; 

Figures 9A-9B are flowchart diagrams illustrating 
another pack transmission operations of the net- 
work switch of Figure 6; and 

Figure 10 is a block diagram illustrating exemplary 
optional dynamic traffic monitoring and load balanc- 
ing features, 

[0017] Refernng now to Figure 1, a simplified block 
diagram of a network system 100 is shown including 
bonded ports according to the present invention be- 
tween two network switches 102, 106. Port bonding in- 
volves bonding two or more ports together into a bonded 
port set anfJ treating the bonded port set as though it 
were a single port, except that traffic flow is allowed on 
each of the bonded ports. A bonded port set of one 
switch is coupled, port by port, to a corresponding bond- 
ed port set of another switch, thereby creating a com- 
bined bonded link or "fat pipe" between the two switch- 
es. The spanning tree algorithm is modified in each of 
the switches 102, 106 to prevent disabling any of the 
bonded ports, and packet looping is avoided by sending 
each packet out only one bonded port. Also, packets re- 
ceived at any bonded port are not forwarded to the other 
bonded ports in the same bonded port set. In this man- 
ner, a bonded link provides a larger bandwidth for com- 



munication between the switches than would othenwise 
be possible without port bonding. 
[0018] In the embodiments described herein, each 
switch nnay support any number of bonded port sets al- 

5 though a bonded link is provided only between two 
switches. In general, each bonded port set may include 
any subset of all the ports up to the full number of switch 
ports. However, a port may be included in only one 
bonded port set. It is noted that the present invention is 

^0 illustrated herein using switches. However, the concepts 
of redundant ports and port bonding is equally applica- 
ble to network communication devices other than 
switches. 

[0019] With reference to Figure 1, the network system 

'5 100 includes four network devices 110, 114, 116, and 
118, which are assigned MAC addresses A, B, C and D, 
respectively. Each of the network switches 1 02 and 1 06 
include a plurality of ports, each for receiving and trans- 
mitting data. In particular, the network switch 102 in- 

20 dudes a plurality of ports 104 and the network switch 
106 includes a plurality of ports 108. The ports 104, 108 
may all be of the same type and operate at the same 
speed. Alternatively, the ports 104, 108 may include a 
combination of different port types and speeds. For ex- 

25 ample, some ports may operate according to Ethernet 
lOBase-T at 10 megabits per second (Mbps) while oth- 
ers operate at 100 Mbps or at both speeds 10/100. The 
present invention is not limited by the particular type or 
speed of the ports. 

30 [0020] Each of the network devices 11 0, 1 1 4, 1 1 6, and 
118 may be personal or laptop computers with network 
interface cards (NICs), sen/ers, data terminal equip- 
ment devices (DTEs), network controllers, or any other 
types of network devices. The network devices 110,114, 

35 116 and 11 8 are coupled to ports 130, 132, 134 and 136, 
respectively, via corresponding links 112. The links 112 
may be any appropriate kinds of transmission media in- 
cluding cables typically used for connecting network de- 
vices in a typical network. For example, the links 112 

40 could be twisted pair cables with RJ-45 connectors if the 
network system 100 is based on the Ethernet architec- 
ture. The links 1 1 2 may also include remote and/or wire- 
less connections. 

[0021] The network system 100 may be a local area 
45 network (LAN), a wide area network (WAN) or any other 
kind of network and may be based on Ethernet, Token 
Ring, ATM or any other network architecture known to 
those skilled in the art. In the embodiment shown, the 
network system 100 is a shared media network that 
so communicates data in the form of packets, where each 
of the devices 110, 114, 116 and 118 and the network 
switches 1 02, 1 06 communicate by sending and receiv- 
ing packets. Each packet may be unicast, multicast, or 
broadcast. Unicast packets typically include a source 
5S identifier or address identifying the sending device and 
a destination identifier or address identifying the intend- 
ed destination device. The identifiers or addresses are 
preferably in the form of a MAC addresses, which are 
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uniquely assigned to identify particular devices. Broad- 
cast and multicast packets include a source address 
and are intended to be broadcast to all other devices In 
the network or to a parttcutar subset thereof. Broadcast 
packets do not include a destinatiori address, but has 
its group bit set to identify it as a broadcast packet. A 
multicast packet provides a source address along with 
one or more destination addresses and also has its 
group bit set. In the present disclosure, multicast and 
broadcast packets are treated the same. 
[0022] The network switch 102 includes port control 
circuitry 1 54 coupled to each of its ports 104 for control- 
ling traffic flow of the ports 104. The network switch 102 
also includes port bonding circuitry 156 coupled to. or 
otherwise incorporated within, the port control circuitry 
1 54. Likewise, the network switch 1 06 includes port con- 
trol circuitry 1 50 coupled to each of its ports 1 08 for con- 
trolling data traffic, and port bonding circuitry 152 cou- 
pled to, or otherwise incorporated within, the port control 
circuitry 1 50. The port control circuitry 1 54. 1 50 manage 
the operation of the respective ports 104, 108 of the 
switches 102, 106 respectively, and the port bonding 
components 156, 152 contain port bonding information 
and facilitate port bonding for the switches 1 02. 1 06, re- 
spectively. Although shown as incorporated within the 
network switches 102, 106, the port bonding compo- 
nents 156, 152 may be external to the switches 102, 106 
in an alternative embodiment. 

[0023] The port control circuitry 154, 150 of each of 
the network switches 102 and 106 examine each packet 
received and performs filtering functions to reduce ex- 
traneous packet generation. In particular, each of the 
port control circuitry 1 54 and 1 50 examines each packet 
and "learns" the source MAC address of each received 
packet and associates or assigns that source address 
with the particular port through which the packet was 
received. Ports of a switch are identified in any desired 
manner, such as a port number. Thereafter, the port con- 
trol circuitry 1 54, 1 50 examines the destination address, 
if any, of each packet and sends that packet to a desti- 
nation port that corresponds to the learned MAC ad- 
dress. For example, when a packet Is sent by the net- 
work device 110 to the port 130 with source MAC ad- 
dress A, the port control circuitry 154 of the network 
switch 102 learns the MAC address A and assigns the 
port number of the port 1 30 to the MAC address A. II the 
received packet includes a destination MAC address B 
designating the device 114, and if the port control cir- 
cuitry 154 has already associated MAC address B with 
port 132, then network switch 102 forwards the packet 
to the port 132. 

[0024] If, however, the destination MAC address for a 
received packet is not yet known to the port control cir- 
cuitry 154, then the received packet is broadcast to all 
the other ports 104 or to all associated ports as further 
described below. Broadcasting the packet in this man- 
ner ensures thai the intended destination device even- 
tually receives the packet. Subsequently, when the net- 



work device associated with the unknown destination 
address responds by sending a packet ot data with its 
own address as the source address in the packet, the 
port control circuitry 154 of the network switch 102 
5 learns the MAC address and assigns it to the port 
through which the packet was received. For example, if 
the MAC address B is not known by the port control cir- 
cuitry 154, the received packet is broadcast to the other 
ports 104 including the port 132. so that the device 114 

10 receives the packet. The device 114 eventually re- 
sponds by sending a packet with source MAC address 
B, so that the port control circuitry 154 iearns MAC ad- 
dress B and assigns the port number of port 1 32 to MAC 
address B. If the packet includes destination MAC ad- 

^5 dress A, then the switch 1 02 does not need to broadcast 
the packet but sends the packet via the port 130. 
[0025] A network switch directly supports only a finite 
number of network devices since it Is physically limited 
to a finite number of ports. Many switches, however, In- 

20 elude one or more uplink ports for connecting to similar 
uplink ports of other network switches to expand the size 
of the network system. Nonetheless, such uplink con- 
nections were limited to the bandwidth of a single port 
since the spanning tree algorithm eliminated redundant 

25 links. 

[0026] The network switch 1 02 includes a bonded port 
set 120 which includes two ports 140 and 142. The port 
bonding circuitry 1 56 facilitates operation to support the 
bonded port set 120. The network switch 106 also has 

30 a bonded port set 122 including two ports 144 and 146, 
where the port bonding circuitry 1 52 supports the bond- 
ed port set 122. The port 142 is coupled to the port 144 
via a link 1 24 and the port 1 40 is coupled to the port 1 46 
via a link 126. The pair of links 124, 126 is collectively 

55 referred to as a bonded link 148. Although the bonded 
link 1 48 shown in Figure 1 couples only two ports of each 
network switch 102, 106, more than two ports could be 
included. If each bonded port of a bonded port set has 
a bandwidth of lOOMbs at half duplex, then due to port 

40 bonding, the aggregate data bandwidth between the 
network switches 102, 106 via the bonded link 148 is 
400Mbs in full duplex mode. The port bonding circuitry 
156, 152 of each of the network switches 102. 106, re- 
spectively, enable communication via the bonded port 

45 link 148 after the bonded link 148 is established. 

[0027] The port control circuitry 154, 150 of each of 
the switches 102, 106, through operation of the port 
bonding circuitry 156, 152, treats its bonded port set 
1 20. 1 22 as though It were a single port. To send a pack- 

50 et from the bonded port set 120, the switch 102 selects 
only one of the bonded ports 140, 142. This selection 
criterion enables the switch 102 to evenly dislnbute traf- 
fic on the bonded port set 1 20 to increase traffic through- 
put. The switch 102 learns or associates MAC address- 

55 es of packets received at Its port of a bonded port set in 
one of two ways. A MAC address is assigned to either 
bonded port 1 40 or 142, or to the entire bonded port set 
120. Such address assignments are primarily for the 
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purpose, of transmission rather than reception of pack- 
ets. If the MAC address is assigned to a particular bond- 
ed port, the assignnnent does not have to be to the same 
port that received the packet. Subsequent packets with 
the same address may be received at any of the ports 
of the bonded port set without effecting the assignment. 
The switch 106, through operation of the port bonding 
circuitry 152, operates in a similar manner. 
[0028] An example of port bonding operation is now 
described. Suppose the network device 110 sends a 
unicast packet to port 130 of the switch 102 with the 
source address A and a destination address C. If the 
switch 102 has not previously learned the address C, i. 
e. does not know the port number associated with the 
destination MAC address C, it broadcasts the packet to 
all the ports 104 other than the received port 1 30. Also, 
the switch 102 selects only one bonded port 140 or 142 
of the bonded port set 1 20 to send the packet. As further 
described below, assignment to the bonded port set or 
to either bonded port describes two different modes of 
operation or two alternative embodiments. One goal of 
either embodiment is to distribute network traffic be- 
tween the bonded ports evenly. In the embodiment 
shown, the switch 1 02 operates to assign MAC address- 
es to bonded port sets in general. Thus, the switch 102 
assigns MAC address C to the bonded port set 1 20 rath- 
er than to either bonded port 1 40 or 142. In this manner, 
the determination of which bonded port 140 or 142 of 
the bonded port set 120 to use is made on a packet by 
packet basis. 

[0029] Since it is typically desired to maintain proper 
ordering of the packets for each source port, however, 
the port control circuitry 154 and the port bonding cir- 
cuitry 1 56 also ensure that the packets are and trans- 
mitted in the same order that they were received by re- 
spective ports. One method to ensure proper ordering 
is that the port control and port bonding circuitry 154, 
1 56 alternatively assigns and distributes the packets be- 
tween the bonded ports 140 and 142. For example, if 
the source address C is assigned by the switch 102 to 
the bonded port set 120, then a first packet with desti- 
nation address C to be sent to the device 11 6 is assigned 
to bonded port 142, while the next packet with destina- 
tion address C is assigned to bonded port 1 40, and so 
on. Assignment forthree or more bonded ports is similar 
where traffic is evenly distributed, such as using a 
round-robin assignment of packets on a packet by pack- 
et basis. 

[0030] The switch 102 also ensures that the packets 
are transmitted in the same order as received. How 
proper transmission order is achieved may depend up- 
on the particular hardware configuration. It is not nec- 
essary to wait for a packet to be completely transmitted 
by one bonded port before sending the next packet on 
the other bonded port. It is sufficient that the switch 102 
begins transmitting a prior packet on one port before in- 
itiating transmission of a subsequent packet on the next 
bonded port. For example, a single queue is defined for 



the entire bonded port set 120 and the packets are 
queued and transmitted in proper order. For the packet 
by packet embodiment, the network switch receiving 
packets on a bonded port set also maintains proper or- 
5 daring of the packets. For example, a first packet re- 
ceived by th e network switch 1 06 on port 1 44 addressed 
to the device 1 1 6 is sent to the device 1 1 6 before a next 
packet received by the port 146 also addressed to the 
device 116. 

10 [0031] Suppose that the network switch 102 selects 
port 142. The switch 106 receives the packet with the 
source address A on the port 144 with the destination 
address C. If the switch 105 has not previously received 
a packet with the MAC address A, it assigns the MAC 
15 address A to the port set 1 22 or to either bonded port 
144 or 1 46. The switch 106 operates to assign address- 
es to a particular bonded port of a bonded port set rather 
than to the bonded port set in general as described 
above for the network switch 1 02. In this alternative em- 
bodiment, the packets to be sent by the network switch 

105 to a device with a given destination address via the 
bonded port set 122 are sent by the same bonded port 
to which that destination address has been assigned. 
This MAC assignment embodiment operates on the as- 
sumption that most devices generate roughly the same 
amount of packets, and has the inherent benefit in that 
packet ordering per source port is automatically main- 
tained. As described further below, a switch operating 
using particular port assignments may dynamically alter 
assignments if traffic load is not distributed evenly. 
[0032] In the embodiment shown, the switch 106 as- 
signs the bonded port 146 to the MAC address A. If the 
network switch 106 does not know the destination MAC 
address C, it broadcasts the packet to all its ports 108 
including the port 1 34 and excluding the ports 144, 146 
of the bonded port set 122. since the bonded port set 
122 is the source "port". The network device 116 thus 
receives the packet with address A. The network device 
116 responds by sending a unicast packet to the switch 

106 with the source address C and destination address 
A. The network switch 106 receives the packet at port 
134 and assigns the MAC address C of the device 116 
to the port 1 34. The network switch 1 06 determines that 
the MAC address A is assigned to the bonded port 146 
and sends the packet to the switch 1 02 via port 1 44. The 
network switch 102 receives the packet at the bonded 
port set 120, assigns MAC address C, and then sends 
the packet via the port 130 for delivery to the network 
device 110. It is noted that any subsequent packets re- 
ceived by the switch 106 at any of the ports 108 other 
than the ports 144 and 146 with destination address A 
is sent via port 146 since assigned to MAC address A. 
[0033] Since the network switch 102 transmits pack- 
ets on a packet by packet basis, subsequent packets 
sent by either device 11 Oor 114 with destinatton address 
C are sent by switch 102 via either bonded port 1 40 or 
142. Consequently, the switch 106 receives packets 
with destination MAC address C at either bonded port 
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144 or 146. The port control circuitry 150 of the switch 
106, in cooperation with the port bonding circuitry 152, 
recognizes the fact that the ports 144 and 146 are in- 
cluded in the same bonded port set 122 and does not 
change the assignment of MAC address C. Also, sup- 
pose that the switch 106 assigns bonded port 144 to 
MAC address B. If device 118 sends two packets, each 
with source address D, including a first packet with des- 
tination address A and a second packet with destination 
address B, then the switch 106 sends the first packet 
via bonded port 1 46 and the second packet via port 1 44 
^to the switch 1 02. The switch 1 02 receives the two pack- 
ets with source address D on two different bonded ports 
140 and 142, respectively. Nonetheless, the port control 
circuitry 154 of the switch 102, in cooperation with the 
port bonding circuitry 156, recognizes the fact that the 
ports 140 and 142 are included in the same bonded port 
set 120 and does not change the assignment of MAC 
address D. 

[0034] Port bonding according to the present inven- 
tion effectively increases the bandwidth between any 
two switches by distributing packets among multiple 
bonded ports rather than a single port. For the network 
system 1 00 shown in Figure 1 , both of the switches 102 
and 1 06 send each packet on either of the two bonded 
ports, which effectively doubles the throughput between 
the two network switches 102, 106 as compared to a 
single uplink port. A bonded port set including three 
ports effectively triples the throughput as compared to 
a single port. 

[0035] The relative increase in bandwidth may de- 
pend upon the selection and assignment criterion. As- 
signment of addresses to bonded port sets in general 
provides the advantage that traffic is distributed on an 
incremental or a packet by packet basis, which assures 
even distribution of traffic. The assignment to particular 
ports provides the advantage that the port bonding cir- 
cuitry 156 or 152 makes an initial selection determina- 
tion, where port selection is primarily performed in hard- 
ware thereafter. Also, packet order per source port is 
automatically achieved in this embodiment without fur- 
ther logic or circuitry. However, specific bonded port as- 
signment may result in uneven distribution in that some 
devices may generate significantly greater amounts of 
traffic than others. In the later case, dynamic monitoring 
and adjustment of assignments is then utilized to modify 
the assignments to obtain relatively equal traffic distri- 
bution among the bonded ports for each bonded port 
set. 

[0036] Port bonding can be used in conjunction with 
the concept of virtual L^N (VLAN), which is employed 
to group ports or to group devices into one or more sub- 
set LANs. VLANs may be defined in several different 
ways, such as a Vl^N bit map or group of bits that rep- 
resent a VLAN. The membership of any specific port in 
a VLAN is represented by a corresponding bit in the 
VLAN bit map, which indicates whether the port is a 
member of that VLAN. Each port includes a VLAN bit 



map identifying VLAN membership. If VLAN capabilities 
are enabled, then a switch sends packets from one port 
in a VLAN only to other ports that are members of the 
same VLAN. In the erfibodiments described herein, 

s VLANs are allowed to overlap, so that one or more ports 
may be included in two or more VLANs. 
[0037] As described further below, VLAN bit maps 
may be used by the port bonding circuitry of a switch 
according to the present invention to assign bonded 

10 ports to MAC addresses for broadcast and destination 
miss packets. A VLAN may include multiple bonded port 
sets and each bonded port set may be included in more 
than one VLAN. If a VLAN includes one bonded port, 
then it effectively includes all of the bonded ports in that 

IS bonded port set. 

[0038] Figure 2 is a perspective diagram of a network 
system 200 with a network switch 204 participating in 
multiple bonded port sets. The network system 200 has 
four network switches 202, 204, 206 and 208, where the 

20 switches 202, 206 and 208 each have one bonded port 
set 210, 216, and 220. respectively, and where the 
switch 204 has three corresponding bonded port sets, 
21 2. 21 4 and 21 8, The switches 202, 204, 206, and 208 
include the controllers and port bonding components 

2S (not shown) as previously discussed with reference to 
Figure 1 . The bonded port sets 21 2, 21 4, and 21 8 of the 
switch 204 are connected to the bonded port sets 210, 
216, and 220, respectively. The bonded port sets 212, 
214, and 218 have four, two and three bonded ports, 

30 respectively. Each bonded port of each bonded port set 
of each switch is linked to a corresponding port of a cor- - 
responding bonded port set of another switch. In partic- 
ular, four separate links of a bonded link 244 couple to- 
gether the four bonded ports of the bonded port sets 

35 210, 212. Likewise, two separate links of a bonded link 
246 couple together the two bonded ports of the bonded 
port sets 214, 216 and three separate links of a bonded 
link 248 couple together the three bonded ports of the 
bonded port sets 218, 220. 

40 [0039] Figure 3 is a diagram of a network system 300 
illustrating that bonded port sets can include ports of dif- 
ferent bandwidths. Network switches 302, 308 include 
one or more ports 304 and 310, respectively, that oper- 
ate at one speed and one or more ports 306 and 31 2, 

^ respectively, that operate at a different and greater 
speed. A bonded port set 31 4 on the switch 302 includes 
ports 320 and 322 from the ports 304 and ports 324 and 
326 from the ports 306. Similarty, the bonded port set 
31 6 on the switch 308 includes ports 330 and 332 from 

50 the ports 310 and ports 334 and 336 from the ports 31 2. 
A bonded link 350 established between the bonded port 
sets 31 4 and 316 includes links 340, 342, 344, and 346 
for coupling together ports 320 and 330, 322 and 332, 
324 and 334, and 326 and 336. respectively The links 

55 340 and 342 are of lower bandwidth than the links 344, 
and 346. 

[0040] The distribution of packets over the bonded 
link 350 is preferably maintained in proportion to the 
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bandwidth capacities of the included ports. For exam- 
ple, if the ports 306. 31 2 operate at ten times the speed 
o1 ports 304. 310, then approximately ten times the 
number of packets are sent across the links 344, 346 
than the links 340, 342. In general, the ratio of the packet 
traffic through each of the ports of a bonded port set cor- 
responds to the ratio of the bandwiSths of the different 
ports of the bonded port set. Such traffic distribution pro- 
vides maximizing the throughput of a network switch 
without causing the overload of any individual link. 
[0041] Figures 4A-4B are flowchart diagrams illustrat- 
ing exemplary packet transmission operations of the 
network switches 102, 106 of Figure 1. As described 
above, the switch 102 assigns bonded port sets in gen- 
eral to MAC addresses whereas the switch 1 06 assigns 
particular bonded ports to each MAC address. The flow- 
chart handles both cases. If VLAN capabilities are im- 
plemented and enabled, then each switch 102, 106 
sends packets only to those ports in the same VLAN 
associated with the source MAC address. If VLAN ca- 
pabilities are not enabled, then packets are forwarded 
to all the other ports in the switch. If bonded ports are 
employed, then such broadcast packets are forwarded 
to only one port among the ports of the bonded port set, 
in addition to all the other (non-originating) ports in the 
switch. 

[0042] Figure 4A describes the normal operation of 
receiving packets and determining if the source MAC 
address is known, and subsequently transmitting the 
packet. At first step 406. the switch receives a packet 
and at next step 408 the switch retrieves the source 
MAC address from the packet. At next decision step 
410, the switch determines if the source MAC address 
is known. The port control circuitry 150, 154 preferably 
includes memory (not shown) for storing MAC address 
and assigned port numbers or identifiers of bonded port 
sets. If the source MAC address is known, operation 
proceeds to next step 41 2 where the switch compares 
the assigned port or bonded port set to the source port 
at which the packet was received. Operation proceeds 
to decision step 414, where it is queried whether the 
source port is the same as the assigned port or if the 
source port is in the same bonded port set as the as- 
signed bonded port set. 

[0043] If the source and assigned port numbers match 
or if the source port is a member of the assigned bonded 
port set as determined at step 414, then operation pro- 
ceeds to step 452, described below. If the source and 
assigned port numbers do not match or if the source port 
is not a member of the assigned bonded port set, then 
the switch determines that the network device associat- 
ed with the source MAC address has moved to a differ- 
ent port. If so, operation proceeds to next step 420, 
where the switch assigns the source port to the MAC 
address if the source port Is not bonded. If the source 
port is bonded, then the switch assigns the MAC ad- 
dress to either a particular bonded port or the bonded 
port set in general. Note that if the MAC address as- 



signed to a specific bonded port, that it does not have 
to be the source port but may be any other bonded port 
in the same bonded port set as the source port. Such 
MAC by MAC address assignment is used in an attempt 
5 to provide even traffic flow among the bonded ports. Al- 
so, VLAN information is updated if included. Operation 
then proceeds to next step 452. 
[0044] Referring back to decision step 410, if it is de- 
termined that the source MAC address is new, then the 

10 switch learns the new MAC address at next step 430 
and updates the VLAN information if necessary. Gener- 
ally, a new entry is made in memory for the new MAC 
address. Operation proceeds to next decision step 432, 
where the switch determines if the source port is bond- 

^5 ed. If the source port is bonded as determined at step 
. 430, the switch proceeds to next step 434 to assign the 
MAC address to either a bonded port or to the bonded 
port set of the source port. Again, if assigned to a par- 
ticular bonded port, the assignment need not be the 

20 source port but may be another bonded port in the same 
bonded port set of the source port. Operation then pro- 
ceeds to step 452. Referring back to step 432, if it is 
determined that the received port is not a bonded port, 
then the source port is assigned to the new MAC ad- 

2S dress at next step 436, and operation proceeds to step 
452. 

[0045] At step 452, the switch determines if the re- 
ceived packet has a un least or broadcast destination 
MAC address. If unicast, operation proceeds to next 

30 step 454 to identify the destination MAC address from 
the packet. At next step 455 the switch determines if the 
destination MAC address is known in a similar manner 
as described above for the source port. If the destination 
MAC address of the packet is known, then operation 

35 proceeds to next step 456 to determine whether the 
MAC address is assigned to a particular port or to a 
bonded port set. If assigned to a bonded port set rather 
than a particular port, then operation proceeds to next 
step 457 where the switch assigns a bonded port for 

40 sending the packet. If the destination address was pre- 
viously assigned to a particular port as determined in 
step 456 or after a bonded port is assigned in step 457, 
operation proceeds to step 458. where the switch deter- 
mines if the source and destination ports are both bond- 

4S ed in the same bonded port set. If the source and des- 
tination ports belong to the same bonded port set, then 
operation proceeds to step 466, where the packet is 
dropped and operation returns to step 406 for the next 
packet. 

so [0046] Referring back to step 458, if the switch deter- 
mines that the destination and source ports are not 
members of the same bonded port set, then the switch 
detenmines whether the VLAN capabilities are enabled 
at next step 460. If VLAN capabilities are not enabled, 

55 then operation proceeds to step 476 where the switch 
transmits the packet to the destination port, and opera- 
tion returns to step 406 for the next packet. If VLAN is 
enabled as determined at step 460, then operation pro- 
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ceeds to next decision step 464 where the switch deter- 
mines it the destination port is a member of the same 
VLAN as the source port. It It is determined that the des- 
tination port is not part of the same VLAN as the source 
port, then operation proceeds to step 466 where the 
switch drops the packet. Otherwise, the switch proceeds 
to step 476 to transmit the packet to the destination port. 
As before, operation then returns to step 406. 
[0047] Referring back to steps 452 and 455, If the 
packet is a broadcast packet or if the destination MAC 
address is not known, then operation proceeds to deci- 
sion step 462 to determine If VLAN Is enabled. If VLAN 
is enabled, then operation proceeds to next step 474 
where the switch assigns a bonded port for each bonded 
port set in the VLAN excluding the bonded port set of 
the source port, If bonded. Operation then proceeds to 
next step 478. where the switch transmits the packet to 
all the ports in VLAN including any assigned bonded 
ports, and operation returns to step 406. In this manner, 
the switch avoids packet looping by sending the packet 
only out one bonded port for each bonded port set. Re- 
ferring back to step 462, if It is determined that VLAN is 
not enabled, operation then proceeds to next step 470, 
where the switch assigns a bonded port for each of the 
bonded port sets in the network switch while excluding 
a bonded port set if the source port is member of that 
bonded port set. Operation then proceeds to next step 
472, where the switch transmits the packet to all ports 
including the assigned bonded ports, and then operation 
returns to step 406 for the next packet. 
[0048] Referring now to Figure 5, a diagram is shown 
of network system 500 illustrating a packet being trans- 
mitted through a bonded set of ports between two net- 
work switches without any duplication of the packet. The 
network system 500 includes two network switches 508, 
5 1 0. The two network switches 508, 5 1 0 Implement port 
bonding while preventing data traffic received on a port 
in a bonded port set from being forwarded to any of the 
other ports In the same bonded port set. This applies to 
both unicast and broadcast traffic. A bonded port set 542 
on the network switch 508 includes two bonded ports 
538, and 540. On the network switch 510, a bonded port 
set 544 includes two ports 534, and 536. The bonded 
port sets 542 and 544 are linked together with bonded 
link 530. 

[0049] A unicast packet 560 received by a network 
switch 508 from a network device 502, addressed to an 
unknown destination is broadcast by the network switch 
508. However the network switch 508 sends the unicast 
packet to all the ports that are not part of a bonded port 
set 542 (and that is not the originating port) and to only 
one of the ports in the bonded port set 542. the network 
switch 508 selects a specific bonded port, such as the 
port 540, for transmitting the unicast packet 560 to the 
network switch 510 over the bonded link 530. 
[0050] The bonded port set 544 on the network switch 
510 receives the packet on one of its bonded ports. The 
network switch 510 broadcasts the packet 560 to all its 



ports excluding the ports belonging to its bonded port 
set 544 if the destination address is unknown. The net- 
work devices 512, 516, 526 and 524, connected to the 
switch 51 0 via ports 548, 550, 552 and 554, respectively, 

5 receive the packet. If the packet 560 were a broadcast 
packet sent to the switch 508 from the network device 
502, the processing of this broadcast packet by the 
switches 508 and 51 0 is similar to the processing of the 
unicast packet with an unknown destination described 

10 above. 

[0051] Referring now to Figure 6, a more detailed 
block diagram is shown of a network switch 600 Imple- 
mented to support bonded ports according to the 
present invention. The network switch 600 could be 

'5 used as either of the switches 1 02 or 1 06 or both in the 
network system 100 of Figure 1, or as the network 
switches 202, 204, 206. 208, 302, 308, 508, 510 de- 
scribed previously. However, the network switch 600 as- 
signs particular bonded ports to each MAC address 

20 rather then dynamically distributing traffic on a packet 
by packet basis as was describe for switch 102 above. 
The network switch 600 implements port bonding using 
a combination of hardware and software. The hardware 
includes a plurality of registers and memory for defining 

25 bonded ports and for assigning MAC addresses to par- 
ticular bonded ports. Software operating on the network 
switch 600 includes a modified spanning tree, assign- 
ment of default VLAN bit maps and dynamic reassign- 
ment of MAC addresses to particular bonded ports. 

30 [0052] In the embodiment shown, the network switch 
600 includes six similar quad controller or quad cascade 
(QC) devices 602, each incorporating four ports 603. 
The QC devices 602 may be implemented in any de- 
sired manner, such as integrated into a single Appllca- 

55 tlon Specific Integrated Circuit (ASIC) package or as 
separate integrated circuit (IC) chips as shown. In the 
embodiment shown, each port 603 operates at 10 Mbps 
at half duplex, for a total throughput of 20 Mbps per port 
at full duplex. Each of the QC devices 602 preferably 

40 includes a processor interface coupled to a QC/CPU 
bus 604, and a bus interface coupled to a high speed 
bus (HSB) 606. The HSB 606 includes a data portion 
606a and various control and status signals 606b. 
[0053] The HSB 606 and the QC/CPU bus 604 are 

^ further coupled to an Ethernet Packet Switch Manager 
(EPSM) 610, which is implemented as an ASIC in the 
embodiment shown, although the present invention is 
not limited to any particular physical or logical imple- 
mentation. The EPSM 610 is further coupled to a mem- 

50 ory 612 through a memory bus 614, which includes a 
data and address portion 61 4a and control signals 61 4b. 
[0054] The HSB 606 is coupled to a Thunder LAN 
(TLAN)port interface (TPl) 520, which is further coupled 
to a peripheral component interconnect (PCI) bus 622 

55 including data and address signals 622a and related 
control and status signals 622b. The PCI bus 622 is cou- 
pled to four TLANs 626, which may be implemented in 
any desired manner. The TLANs 626 are preferably the 
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TNETE100 ThunderLANT" PCI Ethernet™ controllers 
manufaclured by Texas Instruments, Inc. (Tl), where 
each incorporates a port 650. To the EPSM 610, the TP! 
620 operates in a similar manner on the HSB 606 as 
another QC device 602 for Interfacing four ports. Thus, 
the E PSM 61 0 effectively "sees" seven (7) quad port de- 
vices. With respect to the PCI bus 622, the TPI 620 em- 
ulates a standard PCI bus to the degree necessary for 
proper operation of the TLANs 626, v^^hich normally in- 
terface with PCI memory devices. The PCI bus 622 is 
coupled to a processor or central processing unit (CPU) 
630, which is coupled to a local processor bus 632 for 
coupling the CPU 630 to local RAM 634, a local flash 
RAM 636, and if desired, a serial port interface 638. The 
serial port interface 638 is preferably a U ART or the like. 
In the embodiment shown, the CPU is a 32-brt, 33 MHz 
i960RP CPU by Intel, although the CPU 630 may be any 
other suitable processor. 

[0055] The CPU 630 generally handles initialization 
and configuration of the TPI 620 and the EPSM 610 up- 
on power up of the network switch 600. The CPU 630 
also monitors and gathers statistics and also manages 
and controls the functions of the various devices of the 
network switch 600 during operation. The CPU 630 fur- 
ther updates hash table data in the memory 61 2 through 
the EPSM 610, The EPSM 610, however, controls ac- 
cess to the memory 61 2 and performs the DRAM refresh 
cycles thereby removing refresh operations from the 
CPU 630. The CPU 630 also acts as an additional net- 
work port for various purposes, and is often referred 
herein as PORT28. Thus, the ports 603, 650 and the 
CPU 630 collectively incorporate ports 
PORT0-PORT2B, respectively. The CPU 630 is further 
coupled to the EPSM 610 through a CPU bus 61 8, which 
includes an address and data portion 618a and related 
control and status signals 618b. The address and data 
portion 61 8a is preferably multiplexed between address 
and data signals. 

[0056] The hardware of the network switch 600, ac- 
cording to this exemplary embodiment according to the 
present invention, supports port bonding via several 
hardware features, including: a pair of port bonding reg- 
isters 658. 660 provided in the EPSM 610; a VLAN bit 
map included In hash table entries of a hash table 654 
provided within the memory 612; a plurality of default 
VLAN bit maps 652 in the memory 612 including a de- 
fault VLAN bit map for each port PORT0-PORT27; and 
several VLAN enable bits in a Hash Table Definition 
Register 662 located in the EPSM 610. A global enable 
bit 656 is also included, where the VLAN definition for 
each of the ports PORT0-PORT27 is according to a de- 
fault VLAN bit map defined in a VLAN register 664 if the 
global enable bit 656 is set. If the global enable bit 656 
is set, then the default VLAN bit maps 652 and the par- 
ticular VLAN bit maps in the hash table 654 are not used 
or otherwise ignored. 

[0057] The port bonding registers 658. 660 are gen- 
erally in the form of bit maps to identify the ports that are 



members of a different port bonding set. In particular, 
each register 658, 660 includes at least 28 bits (0-27). 
each bit representing one of the ports 602. 650 or the 
ports PORT0-PORT27. For example, setting bits 24 and 
5 25 with the remaining bits cleared defines the ports 
PORT24 and PORT25 as belonging to a bonded port 
set. The register 658 identifies a first bonded port set 
and the register 660 identifies a second bonded port set. 
Although the switch 600 includes only two port bonding 
'0 registers 658. 660 for supporting up to two bonded port 
sets, it is understood thai more port bonding registers 
may be included to define more than two bonded port 
sets if desired. The CPU 630 executes software which 
defines VLAN bit maps during operation and preferably 
IS keeps a copy of the VLAN bit map assignments in the 
memory 634, among other information. It is noted that 
the implementation of the VLANs is optional to the im- 
plementation of the port bonding scheme. As such, the 
network switch 600 is also used to support port bonding 
in networks where VLANs are not implemented. 
[0058] The network switch 600 uses the VLAN bit map 
in each hash entry in the hash table 654 as part of the 
port bondirig implementation. When Vl-ANs are not im- 
plemented, the VLAN bit map in each hash entry is usu- 
ally set to all ones to indicate that broadcast packets are 
sent to all ports in the system and all ports may commu- 
nicate via unicast packets. When VLANs are implement- 
ed, the VLAN bit map in each entry in the hash table 654 
is used by the network switch 600 to identify which ports 
broadcast / multicast / unknown-broadcast packets are 
to be sent when being fonwarded, and to identity w/hich 
ports may communicate using unicast packets. The use 
of the VLAN bit map fields are based on the setting of 
the VLAN Enable bits in the Hash Table Definition Reg- 
ister 662. In the network switch 600, VLANs and port 
bonding may be used concurrently, and the VLAN bit 
maps are preferably used for port bonding even if 
VLANs are not needed or defined. Regardless of wheth- 
er VLANs are used or not, when bonded ports are de- 
fined and used, the VLAN bit maps in the hash table 
entries are used to ensure that packet fonwarding loops 
do not occur among ports in the same bonded port set. 
For bonded ports, the network switch 600 sets only one 
of the bits representing a port in a bonded port set in the 
VLAN bit map in each entry in the hash table. 
[0059] To aid in broadcast mapping to evenly distrib-- 
ute broadcast traffic among the ports in a bonded port 
set, the CPU 630 via software sets different port bits in 
each VLAN bit map field for all hash entries v/here the 
bonded port set is included in the VLAN bit map. In this 
way, when broadcast traffic is sent to the bonded port 
set, it gets randomly distributed based upon the source 
address among the ports in the bonded port set to help 
evenly distribute the traffic load. Doing this will help the 
switch achieve as close to full bandwidth potential as 
possible on the bonded port set. The default VLAN bit 
maps are a series of bit maps for each port and are used 
in a similar manner as the hash entry VLAN bit maps 
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except they are used when a packet with an unknown 
source MAC address is received on a port. The set bits 
representing bonded ports are distributed in a similar 
manner, but are distributed among all the ports or 
among all the ports in the VLAN. s 
[0060] Figure 7A is a block diagram illustrating the or- 
ganization of the memory 612 of Figure '6. The memory 
612 is used for several purposes including the storage 
of the hash table 654, the storage of packets of traffic 
data and storage of the per port default VLAN bit maps io 
652 for the ports of the network switch 600. The memory 
61 2 is preferably divided into three main sections includ- 
ing the hash table 654, the default VLAN bit maps 652 
and a packet memory section 704, The hash table 654 
serves as a network device identification section for 
identifying one or more network devices coupled directly 
or indirectly to the network switch 600 via the ports 
PORT0-PORT27. The size of the hash table 654 may 
be programmable based on the number of devices and 
associated addresses and entries desired. The hash ta- 20 
ble 654 may be located anywhere in the memory 612, 
but is preferably located at the beginning of the memory 
612 in the embodiment shown. The default VLAN bit 
maps 652 are preferably stored adjacent or near the 
hash table 654. The size of the packet memory section 2S 
704 is preferably the balance of the remaining memory 
612 not used by the hash memory section 654 and the 
default VLAN bit maps section 652. 
[0061] Figure 7B is a block diagram of the organiza- 
tion of the hash table 654 of the memory 612. The hash 3o 
table 654 is divided into two sections including a first 
primary hash entry section 706 for primary hash entries 
and a second chained hash entry section 708 for 
chained hash entries. In the embodiment shown, each 
of the sections 706, 708 includes up to 8K entries, each 35 
16 bytes in length. 

[0062] Figure 7C is a diagram illustrating the organi- 
zation of a hash table entry 710 representative of each 
of the entries in the hash table 654, including both the 
primary hash entry section 706 and the chained hash 40 
entry section 708. Each entry 710 corresponds to one 
MAC address of network devices coupled to the network 
switch 600. Each of the primary entries are located at a 
hash key, which is determined by "hashing" the MAC 
address for that device. In particular, each network de- 45 
vice is assigned a 48-bit MAC hardware address, which 
is a unique numerical value assigned to each network 
device during the manufacturing process or by setting 
jumpers or switches during network installation. One 
part of this MAC address is assigned to the manufactur- so 
er by the IEEE (Institute of Electrical and Electronics En- 
gineers) and is common to all components from that 
manufacturer; the second part of the hardware address 
is a unique value assigned by the hardware manufac- 
turer. 55 
[0063] The first 6 bytes, or bytes 5-0, of the hash table 
entry 710 contains the MAC address of the device as- 
sociated with that entry. The network switch 600, there- 



fore, adds a hash table entry for each network device 
that sends a data packet Including its source MAC ad- 
dress. 

[0064] The next byte (6) of the hash table entry 710 
contains a binary port number (PortNum) identifying the 
associated port number to which the device is connect- 
ed, where the port number for PORTO is zero, the port 
number for PORT1 is one, the port number for PORT28 
(for the CPU 630) is 28, etc. The next byte (7) is a control 
and age information byte (Control/Age) including a valid 
bit (VALIDENTRY) identifying whether the entry is valid 
or not. where a logic M " indicates the entry is valid and 
logic "0" indicates that the entry is not valid, othenwise 
called an empty entry. The Control/Age byte includes a 
binary age number (AGE) representing the elapsed time 
from the last source access associated with this device. 
[0065] The next four bytes (11:8) define a 29-bit virtu- 
al-LAN (VLAN) bit map value representing port group- 
ings, if used. Each bit of the VLAN value corresponds 
to a respective one of the ports and is set if the device 
or port is grouped with that port, VLAN bit maps enable 
the ports PORT0-PORT28 of the network switch 600 to 
be grouped in any desired combination to form a plural- 
ity of different LANs. For example, if the first five ports 
port PORT0-PORT4 are grouped together, then the 
VLAN value for each hash entry to be included in the 
group is 0000001 Fh, where "h" denotes a hexadecimal 
value. A VLAN value of all ones or 1 FFFFFFFh denotes 
no VLAN groupings for that device. It is noted that it is 
possible for one device to be associated with more than 
one group. For each VLAN bit map, the bit representing 
the port itself (as source port) may or may riot be set 
depending upon the embodiment. The network switch 
600 includes logic to ensure that a packet received by 
a port is not sent back out that port, so that its VLAN bit 
in its VLAN bit map may be set if desired. Alternatively, 
if a network switch relies on the VLAN bit maps for for- 
warding packets, it is desired that the VLAN bit repre- 
senting the source port not be set. 
[0066] The last four bytes (15:12) of each hash table 
entry 71 0 is a link address (Link A[31 :0] or Link Address) 
pointing to the next entry having an identical hash key, 
if any, In the chained hash entry section 708. MAC ad- 
dresses are hashed according to one of several algo- 
rithms. For example, two portions, such as 13-16 bits, 
of each MAC address are logically combined or com- 
pared to calculate a corresponding hash key. Each por- 
tion is combined using exclusive-OR (XOR) logic in a 
bit-wise fashion to form a 1 3-16 bit hash key It is under- 
stood that many other different hashing algorithms are 
known and may be used to combine any particular com- 
binations of the address bits as known to those skilled 
in the art, and that the present invention is not limited to 
any particular hashing scheme. The hash key is used 
as the actual address or as an offset address to locate 
a corresponding hash entry of the primary hash entry 
section 706. Although the MAC addresses are unique, 
the hash key may not be unique such that two different 
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MAC addresses hash to the same hash key. 
[0067] The chained hash entry section 708 is provid- 
ed to store MAC addresses with duplicate hash keys for 
different devices. The next entry is stored in the next 
available location in the chained hash enlr^ section 708. s 
In this manner, if two MAC addresses of two different 
devices hash to the same hash key,'Ihe first or "primary" 
entry is stored in the primary hash entry section 706, 
and the second entry is stored in the chained hash entry 
section 708, and the Link Address of the primary entry io 
points to the second entry If another MAC address 
hashes to the same hash key as the first two, then each 
additional entry is stored in the chained hash entry sec- 
tion 708 and linked together in consecutive order using 
the Link Addresses. Thus, the first points to the second, '5 
the second points to the third, and so on. Each entry 
follows the format of the hash table entry 710. The for- 
mat of the Link address may be defined in any conven- 
ient manner. The Link address typically includes a base 
address portion pointing to the hash memory section 
702 within the memory 612, and an offset portion to the 
actual entry within the hash memory section 702. The 
lower address bits may be set to zero for byte alignment 
as desired. 

[0068] In the preferred embodiment, the last entry in 25 
each chain has its link address set to point to the begin- 
ning of the default VLAN bit maps 652. In this manner. 
If the link address equals the beginning address of the 
default VLAN bit maps 652, then a hash miss on the 
source MAC address has occurred. In that case, the 30 
VLAN bit map in the default VLAN bit maps 652 corre- 
sponding to the source port is used as the VLAN bit map 
for the current packet to determine what ports to send 
the packet to. Since the MAG address causes the CPU 
to enter the new MAC address into the hash table 654, 35 
subsequent packets with the same source MAC ad- 
dress use the VLAN in the hash table 654. 
[0069] In general, the bit corresponding to the port 
number of a bonded port is set for the default VLAN bit 
maps of each bonded- port and the VLAN bit maps of 40 
MAC entries assigned to that bonded port, if any The 
bits for the other bonded ports in the same bonded port 
set are cleared. For each default VLAN bit map of bond- 
ed ports, the bits corresponding to other ports in the 
same VLAN are also set if the VLAN includes the same 
bonded port set. The VLAN bit maps for entries in the 
hash table 654 corresponding to MAC addresses re- 
ceived at bonded ports also have the bits set corre- 
sponding to other ports in a VLAN if that VLAN includes 
the corresponding bonded port set. so 
[0070] It Is noted that VLANs may overlap, and that 
two or more VLANs may include the same bonded port 
set. When a bonded port set is included in multiple 
VLANs, the default VLAN bit maps for the bonded ports 
of the bonded port set includes all the ports of alt the ss 
VLANs in which the bonded port set participates. Since 
the VLAN bit maps for the ports of the bonded port set 
includes all the ports from all the VLANs in which a bond- 



ed port participates (with modifications for the bits of the 
bonded port set), broadcast traffic received over a bond- 
ed port Is sent to all those VLANs. 
[0071] Figure 8 Is a flowchart diagram Illustrating an 
exemplary initialization and reprogrammlng operation of 
the network switch 600 of Figure 6. The network switch 
600, In the present invention, is initialized with default 
settings for the various parameters such as the VLAN 
membership and port bonding information. The network 
switch 600 optionally monitors the traffic through the 
ports, particularly the traffic on bonded ports, and dy- 
namically reassigns VLAN bits to balance traffic if de- 
sired. 

[0072] With reference to Figure 8, the network switch 
600 of Figure 6 detects an input from a user to indicate 
setting initialization values or for reprogramming those 
values at first decision step 804. It is noted that an ini- 
tialization or driver routine may be executed by the CPU 
630 upon power up or reset of the network switch 600. 
where the routine Is stored In non -volatile memory, such 
as the Flash RAM 636. Also, a driver routine may be 
loaded in the memory 634. A user programs the network 
switch 600 via the software port 638, or by executing a 
management program externally and accessing the 
CPU 630 via any one of the ports PORT0-PORT27. In 
this manner, the network switch 600 may be pro- 
grammed initially with bonded port and VLAN assign- 
ments and may be programmed at any time during op- 
eration. 

[0073] If, at the step 804. the network administrator or 
manager indicates the desire to Initialize or reprogram 
the network switch 600, operation proceeds to next step 
805 to receive bonded port set assignments and then to 
step BOB to receive VLAN assignments. The CPU 630 
receives the bonded port set assignments and pro- 
grams the port bonding registers 658, 660 accordingly 
The CPU 630 then receives the enable bit assignments 
in next step 810 Including VLAN enable bits and the glo- 
bal enable bit 656. The enable bits Include a VLAN 
Group Broadcast Enable Bit, a VLAN Miss Broadcast 
Enable Bit. and a VLAN Unlcast Enable Bit When set 
to one. the VLAN Group Broadcast Enable Bit causes 
the network switch 600 to not forward broadcast and 
multicast packets to ports whose bits are set to zero In 
the VLAN bit map field of the hash table entry of the 
packet's source MAC address. If the source MAC ad- 
dress Is not in the hash table 654, the default VLAN bit 
map for the source port is used. When set to one. the 
VLAN Miss Broadcast Enable Bit causes the network 
switch 600 to not broadcast packets, due to destination 
address hash misses, to ports whose bits are set to zero 
in the VLAN bit map field of the hash table entry of the 
packet's source MAC address. If the source MAC ad- 
dress Is not In the hash table 654. the default VLAN bit 
map for the source port is used. When set to one. the 
VLAN Unicast Enable Bit causes the network switch 600 
to not forward unlcast packets to ports whose bits are 
set to zero in the VLAN bit map field of the hash table 
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entry of the packet's source MAC address. If the source 
MAC address is not in the hash table 654. the default 
VLAN bit map for the source port is used. 
[0074] From step 810; operation proceeds to step 
812. where the default VLAN bit maps 652 for all the 
ports are programmed according to the VLAN assign- 
ments received at step 808. If bonded port set assign- 
ments are received at step 806, however, then the bits 
corresponding to the bonded ports in each VL^N are 
assigned. This is shown at next step 81 4, where the net- 
work switch 600 distributes bits corresponding to bond- 
ed ports among the VLAN default bit maps for each 
VLAN. Only one of the bonded port bits in each default 
VLAN bitmap is set. so that bonded ports in each VLAN 
are distributed on a port by port basis in the per port 
default VLAN bit maps 652. 

[0075] As an example, if a VLAN includes ports 
PORT0-PORT4 and PORT24 where PORT24 is bonded 
to ports PORT25 and PORT26, then the VLAN effec- 
tively includes the bonded port set including ports 
PORT24-PORT26. The default VLAN bit map for each 
of the ports PORT0-PORT4 includes all of the non- 
bonded ports PORT0-PORT4 plus one of the bonded 
ports. The VLAN bit map for a given port may or may 
not be defined to include that port. The VLAN bit maps 
described herein include its own port, it being under- 
stood that packets are generally not fonwarded back to 
the receiving port. Separate logic in the network switch 
600, for example, ensures that packets are not forward- 
ed to the receiving port. However, if such logic were not 
provided, the VLAN bit maps for a given MAC address 
would not have its source port set. The default VLAN bit 
map for port PORTO may include ports PORT0-PORT4 
and PORT24 (or. alternatively, may exclude PORTO), 
the default VLAN bit map for port PORT1 may include 
ports PORT0-PORT4 and PORT25 (or. alternatively, 
may exclude PORT1 . etc.). the default VLAN bit map for 
port P0RT2 may include ports PORT0-PORT4 and 
PORT26. the default VLAN bit map for port PORT3 may 
include ports PORT0-PORT4 and PORT24 and the de- 
fault VLAN bit map for port PORT4 may include ports 
PORT0-PORT4 and PORT25. In this manner, the three 
bonded ports PORT24-PORT26 are distributed among 
the five non-bonded ports PORT0-PORT4. 
[0076] Operation proceeds to next decision step 81 5 
to determine whether there are any entries in the hash 
table 554 and if so, whether those entries are to be re- 
programmed. If the network switch 600 is reset or oth- 
erwise being initialized for operation, then there will be 
no hash table entries. However, if the network switch 
600 is being programmed during operation after several 
MAC addresses have been learned and stored in the 
hash table 654, then there are at least two options cor- 
responding to two different embodiments. In a first em- 
bodiment, the entries in the hash table 654 are ignored 
and the entire hash table 654 is initialized as if no MAC 
addresses have been received, and operation proceeds 
from step 81 5 to step 81 8, described below. The network 



switch 600 learns all the MAC addresses again in sub- 
sequent operation. This may be desirable since repro- 
gramming may correspond to significant changes of the 
network system. 
5 [0077] Alternatively, if hash entries exist in the hash 
table 654 and it is desired to keep them, operation pro- 
ceeds from step 815 to step 816, where the network 
switch 600 distributes and reprograms VLAN bit maps 
of each of the entries in the hash table 654. In this case, 

10 however, the bonded ports are distributed in the VLAN 
bit maps in the hash table entries on a MAC address by 
MAC address basis. In the example above for a VLAN 
including ports PORT0-PORT4 and a bonded port set 
including ports PORT24-PORT26, assume that three 

'5 devices with corresponding MAC addresses A, B and C 
are coupled to ports PORT2, PORTS and PORT4. re- 
spectively, and that the hash table 654 includes entries 
for each. The VLAN bit map of the entry for MAC ad- 
dress A may include ports PORT0-PORT4 and 

20 PORT24, the VLAN bit map of the entry for MAC ad- 
dress B may include ports PORT0-PORT4 and 
PORT25, and the entry for MAC address C may include 
ports PORT0-PORT4 and PORT26. This is done in an 
attempt to achieve even distribution assuming the de- 

25 vices in the network system generate approximately the 
same amount of packets over time. 
[0078] At a next step 818, the enable bits are set as 
per the user's preference that were previously specified 
at the block 810. Finally, step 820 is included to illustrate 

20 that the spanning tree information used to establish 
routes between network devices is modified to incorpo- 
rate changes in the bonded port information. Operation 
returns to step 804 until reprogramming, reset or initial- 
ization occurs. 

55 [0079] Figures 9A-9Bare flowchart diagrams illustrat- 
ing packet transmission operation of the network switch 
600 of Figure 6. Figure 9A describes the normal opera- 
tion of receiving packets and looking up the source MAC 
address in the hash table 654 and subsequently man- 

40 aging traffic flow through the bonded ports. Figure 9B 
describes determination of packet type, destination ad- 
dress hashing and lookup and transmission of the pack- 
et. At first step 906, the network switch 600 receives a 
packet and at next step 908, the EPSM 610 hashes the 

^ source MAC address and performs a hash table lookup 
in the hash table 654. At next decision step 910, the 
EPSM 610 determines if the source MAC address was 
found in the hash table 654. If the source address was 
found, operation proceeds to step 91 2, where the EPSM 

50 61 0 compares the assigned port number in the hash ta- 
ble 654 corresponding to the source MAC address with 
the port number of the actual source port. At next step 
914, the EPSM 610 determines if the source and as- 
signed port numbers match. If the source and assigned 

55 port numbers match, then operation proceeds to step 
952, described below. The VLAN bit map in the entry in 
the hash table 654 is used tor transmission, as de- 
scribed below. 
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[0080] If the source and assigned port numbers do not 
match as determined in step 914, operation proceeds 
to step 916 to compare the port bonding registers 65B, 
660 with the source and assigned port numbers. At next 
step 916, it is queried whether the source and assigned 
port numbers are bonded together in the same bonded 
port set. If so, then operation proceeds to step 952. Oth- 
erwise, if the source and assigned ports are not bonded 
together as determined In step.91 8. then operation pro- 
ceeds to step 920 where the default VLAN for the new 
source port is retrieved. Alternatively, the default VLAN 
bit map is not retrieved and the VLAN bit map in the hash 
table 654 is used. In either case, although the MAC ad- 
dress is known, the device associated therewith has 
moved to another port and the contents of the VLAN bit 
map in the hash entry is no longer valid. From step 920, 
operation proceeds to step 932, described below. 
[0081] Normally if the source and assigned port num- 
bers do not match, the E PSM 61 0 generates an interrupt 
to the CPU 630 to indicate that the device associated 
with the source MAC address has moved to a different 
port. The interrupt to the CPU 630 is known as a "Port 
Miss' interrupt. However, under the bonded port ar- 
rangement, packets from the same MAC address will 
typically get evenly received on the bonded ports within 
the same bonded port set and therefore, the "Port Miss" 
interrupt feature is effectively disabled for packets re- 
ceived on ports within the same bonded port set. This 
is a function of the port bonding registers 658, 660. 
Thus, if the source and assigned ports do not match, 
then the EPSM 610 checks each port bonding register 
658, 660 at step 916 to determine if both bits that rep- 
resent the source port and the previously assigned port 
in the hash entry for the source MAC address are set. 
If both bits are set in the same port bonding register, 
then the source and assigned ports are bonded and an 
interrupt to the CPU 630 is not generated. 
[0082] Referring back to the decision step 910, if the 
source MAC address is new and is not located in the 
hash table 654, then operation proceeds to step 930 
where an interrupt is generated to the CPU 630 to create 
a new hash entry in the hash table 654 for the new MAC 
address. As described previously, if a hash miss occurs, 
the Link address points the hardware to the beginning 
of the default VLAN bit maps 652. The default VLAN bit 
map is used to determine which ports are allowed if 
VLAN is enabled. From step 930 or 920, operation pro- 
ceeds to decision step 932. where it is determined 
whether the source port is bonded. If the source port is 
bonded, operation proceeds to step 934 where the CPU 
630 assigns one of the bonded ports of the bonded port 
set to the MAC address. The assigned bonded port is 
not necessarily the source port at which the packet was 
received, but instead the assignment is made to balance 
traffic on the bonded port set. Bonded port numbers are 
assigned to MAC addresses in a round-robin fashion, a 
random fashion or in any other fashion todistribute MAC 
addresses evenly (or to achieve proportional band- 



width) across the ports of the bonded port set. The VLAN 
bit map of the new or modified entry in the hash table 
654 is programmed according to the bonded port as- 
signment. In particular, the bits in the VLAN bit map cor- 
5 responding to ports in the same VLAN or VLANs are set. 
From step 934, operation proceeds to step 942, where 
it is determined if any one or more bonded ports are in 
the default VLAN bit map of the source port, as further 
described below. 

10 [0083] Referring back to step 932. if it is determined 
that the source port is not a bonded port, then the port 
number of the actual source port is assigned to the MAC 
address and the hash table entry is updated at step 940. 
Operation proceeds to next step 942, where it is deter- 

'5 mined if any one or more bonded ports are in the default 
VLAN bit map of the source port. If not, operation pro- 
ceeds to step 952. If the default VLAN bit map includes 
one or more bonded port sets even though the source 
port is not bonded itself, operation proceeds to step 944. 

20 where the software assigns one of the bonded ports to 
the MAC address. This is achieved by setting the corre- 
sponding bit in the VLAN bit map of the entry in the hash 
table 654 in a similar manner as described previously to 
achieve even distribution of traffic. Subsequently oper- 

2S ation proceeds to the next step 952. 

[0084] At step 952 in Figure 9B. it is determined 
whether the received packet is a broadcast or multicast 
packet. If not, then the packet is un least and operation 
proceeds to step 954 to lookup the destination MAC ad- 

30 dress in the hash table 654. At the next step 956, It is 
determined whether the destination MAC address is 
found in the hash table 654. If found, then at next step 
958. the EPSM 61 0 determines if the VLAN Unicast En- 
able Bit is set. If not set, then the network switch 600 

35 transmits the unicast packet to the destination port at 
next step 964 and operation returns back the step 906. 
If the VLAN Unicast Enable Bit is set as determined in 
step 958, then at next step 960, it is determined whether 
the destination port is part of the VLAN of the source . 

40 port. If the source MAC address was found and the 
source port matched or was in the same bonded port 
set as the assigned port, then the VLAN bit map in the 
hash table 654 is used, but otherwise, the default VLAN 
bit map for the source port is used. If it is determined at 

4S step 960 that the destination port is not part of the VLAN 
of the source port, then operation proceeds to step 962, 
at which the network switch 600 drops the packet. From 
step 962, operation returns back to step 906 for the next 
packet. If the destination port is part of the VLAN of the 

50 source port, then operation proceeds to step 964, at 
which the network switch 600 transmits the unicast 
packet to the destination port, and operation returns to 
step 906. 

[0085] Referring back to step 956. if the destination 
55 MAC address was not found in the hash table 654, then 
operation proceeds to decision step 966 to determine if 
the VLAN Miss Broadcast Enable bit is set. If not, then 
operation proceeds to step 968 where the network 
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switch 600 transmits the packet from alt of the non-bond- 
ed ports and one bonded port lor each bonded port set 
defined by the port bonding registers 658, 660, and op- 
eration returns to step 906. The selected ports may be 
predefined or determined dynamically at the time step ' s 
968 is performed. In the preferred embodiment, the 
VLAN Miss Broadcast Enable bit is'"set' if port bonding 
is enabled, so that the network switch 600 simply trans- 
mits that packet from all other ports regardless of the 
bits in the port bonding registers 658, 660 as port bond- io 
ing is not enabled. 

[0086] If the VLAN Miss Broadcast Enable bit is set 
as determined at step 966, then operation proceeds to 
decision step 970, where it is queried whether the 
source MAC address was found. If so, operation pro- '5 
ceeds to step 972 to access the VLAN bit map from the 
source hash table entry. Otherwise, if the source MAC 
address was not found, operation proceeds to step 974 
to access the default VLAN bit map for the receiving 
port. From either steps 972 or 974. operation proceeds 20 
to step 976, where the network switch 600 transmits the 
packet from those ports indicated in the retrieved VLAN 
bit map. This VLAN bit map is either the VLAN bit map 
in the hash table 654 or the default VLAN bit map if the 
MAC address was new or moved to a different port. 2S 
From step 976, operation returns to step 906 for the next 
packet. 

[0087] Referring back to the step 952, if the packet is 
determined to be a broadcast packet, then it is deter- 
mined whether the VLAN Group Broadcast Enable bit so 
is set at a next step 978. If the VLAN Group Broadcast 
Enable bit is not set, operation proceeds to step 968 de- 
scribed above and if set, operation proceeds to step 970 
also described above. A broadcast or multicast packet 
is treated in a similar manner as a unicast packet with 35 
unknown destination port. 

[0088] Figure 10 is a block diagram illustrating exem- 
plary optional dynamic traffic monitoring and load bal- 
ancing features. As described above, the network 
switch 600 assigns a bonded port to each MAC address 40 
"residing on" a bonded port set or received by a bonded 
port in that bonded port set. Thus, unicast traffic desig- 
nating a MAC address residing on a bonded port set as 
the destination is always sent to the assigned bonded 
port. Although the assignments are evenly distributed, ^ 
they are randomly or arbitrarily made and some devices 
in operation may be significantly more active than oth- 
ers. For example, a MAC address residing on a bonded 
port set may correspond to a server, where most of the 
traffic from the network switch 600 is to that server This so 
may cause an imbalance of the traffic on the bonded 
port to which the server is assigned. In that case, it may 
be desired to move other MAC addresses from that 
bonded port to other bonded ports of the same bonded 
port set. Such reassignment should consider the relative ss 
bandvyidth of the bonded ports. If bandwidth is equiva- 
lent, then traffic flow should be as equal as possible. If 
the bandwidth of two or more bonded ports are not 
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equal, then the speed or bandwidth ratio of the bonded 
ports is considered. 

[0089] In a similar manner, one or more devices cou- 
pled to non- bonded ports of the network switch 600 may, 
over time, generate a significantly greater amount of 
broadcast traffic than other devices coupled to other 
non-bonded ports of the network switch 600. Regard- 
less of whether VLAN assignments are used or not, the 
bonded port assignments of several devices of a given 
bonded port set may, albeit evenly distributed on a MAC 
by MAC address basis, result in uneven broadcast traffic 
flow on the bonded ports of the bonded port set. The 
VLAN bit map settings are adjusted to redistribute 
broadcast traffic to achieve more even traffic flow. 
Again, the relative bandwidth of bonded ports should be 
considered when determining reassignments. 
[0090] At step 1002, the CPU 630, operating in con- 
junction with the EPSM 610, continuously gathers sta- 
tistical information of the ports PORTO -PORT27 to eval- 
uate traffic patterns for both unicast and broadcast pack- 
ets, especially through the bonded ports of each bonded 
port set. Such monitoring is made on a MAC address by 
MAC address basis to determine the amount of traffic 
associated with each MAC address. Such monitoring al- 
so distinguishes between unicast and broadcast traffic 
to determine the type of adjustments to be made. 
[0091] From step 1002. operation proceeds to two dif- 
ferent steps based on the type of adjustments that need 
to be made. Both paths are performed throughout oper- 
ation of the network switch 600 to balance unicast and 
broadcast traffic. In one path, at a next step 1008. the 
CPU 630 calculates the ratio of unicast traffic transmit- 
ted by each of the bonded ports of each bonded port set 
and then compares the calculated ratio to the ratio of 
the bandwidths of the ports of the bonded port at a step 
1 01 0. If the traffic ratio is found to be similar to the ratio 
of the bandwidths, the network switch 600 proceeds to 
optional step 1012 to determine whether more band- 
width is needed or desired. This determination may be 
made in any one of several ways, such as monitoring 
the queues for the bonded ports or counting the number 
of dropped packets, if any If it is desired to increase 
bandwidth, operation proceeds to step 1014, where the 
network administrator or manager is alerted. Such alert 
is made in any one or more of many different ways, such 
as a message or flag set within the network switch 600 
or sent to a management console. The network admin- 
istrator may respond by adding ports to one or more 
bonded port sets to relieve the bandwidth problem. 
From step 1014 or from step 1012 if bandwidth is suffi- 
cient, operation returns to step 1002. 
[0092] Referring back to the step 1010, if the CPU 630 
determines that the unicast traffic ratio of the bonded 
ports is not comparable to the bandwidth ratio of the 
ports, then operation proceeds to step 1016, where the 
bonded ports of MAC addresses associated with the 
bonded port set are reassigned to balance traffic. In par- 
ticular, the CPU 630 reassigns the bonded ports in the 
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hash table entries to achieve a more even distribution Cli 
of traffic according to the statistical data gathered at step 
1002. Operation then proceeds to optional step 1012 to 1. 
determine if more bandwidth is necessary. 
[0093] In a second path, the CPU 630 calculates the s 
ratio of broadcast traffic of the bonded ports at a step 
1018 from step 1002. At next stepT020, the CPU 630 
determines whether VLAN bit map reassignment should 
be performed to balance the broadcast traffic through 
one or more of the bonded port sets. If the VLAN bit io 
maps do not need reassignment, operation proceeds to 
step 1012, described above. It it is determined at step 
1020 that reassignment is desired to more evenly dis- 
tribute broadcast traffic, operation proceeds to next step 
1 022 to reassign VLAN bit map settings of bonded ports, is 
Operation then proceeds to step 1012 as described 
above. 

[0094] It this manner, traffic through the bonded ports 

is optionally monitored and the MAC address assign- 2. 

ments and/or VLAN bit map assignments are dynami- 20 

cally modified during operation to more evenly distribute 

traffic. 

[0095] It is appreciated that a network communication 
device with bonded ports according to the present in- 
vention enables a higher bandwidth capability between 2S 3. 
any two devices as compared to a single uplink connec- 
tion. Any number of bonded ports may be included in a 
bonded link between two devices to Increase the avail- 
able bandwidth by a proportional amount. Each device 
treats its bonded port set as though it were a single port. 30 
Thus, to avoid packet looping, each packet is sent via 4. 
only one of the bonded ports and packets received at 
one bonded port are not re -transmitted on another bond- 
ed port of the same bonded port set. To achieve the high- 
est bandwidth possible on the bonded link, each com- 35 
munication device distributes traffic on each bonded 
port as evenly as possible. The traffic distribution de- 
pends on the selection criterion for selecting bonded 
ports, where any one of several methods may be used, 
such as random, round-robin, etc. The selection of a 40 
bonded port to send a packet is made either on a packet 5. ' 
by packet basis or by source identifier (f^AC address) 
assignment as previously described. In the assignment 
embodiment, the traffic may be dynamically monitored 
and the assignments periodically adjusted to achieve 
the highest bandwidth on each bonded link. 
[0096] Although a system and method according to 
the present invention has been described in connection 6. 
with the preferred embodiment, it is not intended to be 
limited to the specific form set forth herein, but on the so 
contrary, It is intended to cover such alternatives, mod- 
ifications, and equivalents, as can be reasonably includ- 
ed within the spirit and scope of the invention as defined 
by the appended claims. 



A network communication device, comprising: 

a plurality of ports, by and from which packets 
of data are received and sent; and port control 
circuitry coupled to said plurality of ports, com- 
prising: 

a port manager that directs packets be- 
tween said plurality of ports, and 
port bonding circuitry coupled to said port 
manager that bonds at least two of said plu- 
rality of ports together into a bonded port 
set, and. for each packet to be sent via said 
bonded port set, selects one bonded port 
of said bonded port set. 

The network communication device of claim 1, 
wherein said port bonding circuitry includes at least 
one port bonding register that identifies which of 
said plurality of ports are bonded in said bonded 
port set. 

The network communication device of claim 1 or 
claim 2, wherein said port bonding circuitry selects 
a bonded port on a packet by packet basis so as to 
achieve a relatively even distribution of packets 
sent by bonded ports of said bonded port set. 

The network communication device of any of claims 
1 to 3, wherein: 

each of said packets includes source identifi- 
ers; and further comprising 
memory coupled to said port control circuitry 
that stores a plurality of entries, each of said 
plurality of entries including a unique source 
identifier and a corresponding port identifier. 

The network communication device of claim 4, said 
port bonding circuitry further comprising, for each 
entry in said memory that includes a source identi- 
fier that was received at any bonded port of said 
bonded port set, programming said corresponding 
port identifier to represent said bonded port set. 

The network communication device of claim 4, said 
packets including unicasl packets having a destina- 
tion identifier, wherein 

for each entry in said memory that includes a 
source identifier that was received at any bond- 
ed port of said bonded port set, said port bond- 
ing circuitry programs said port identifier to 
identify one bonded port of said bonded port set 
so as to achieve a relatively even distribution 
of source identifiers among said bonded ports 
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of said bonded port set; and 
said port manager uses said progrannmed port 
identifier that corresponds to a destination 
identifier ofa unicast packet to select a bonded 
port to send said unicast packet via said bond- s 
ed port set. 

7. The network communication device of claim 6, 
wherein 

said port bonding circuitry monitors unicast io 
packets sent via each bonded port of said bonded 
port set and periodically re-programs said port iden- 
tifier of each of said plurality of entries so as to 
achieve a relatively even distribution of unicast 
packet traffic among the bonded ports of said bond- 
ed port set. 

8. The network communication device of claim 4, 
wherein 

20 

each of said plurality of entries in said memory 
that has a source identifier that was received at 
any non-bonded port of said plurality of ports 
further includes a bit map having a bit corre- 
sponding to each of said bonded ports; and 2S 
said port bonding circuitry programs each bit 
map of each of said plurality of entries to assign 
one bonded port ot said bonded port set so as 
to achieve a relatively even distribution of bond- 
ed port assignments to source identifiers cor- ^ 
responding to non-bonded ports. 

9. The network communication device of claim 8, 
wherein 

35 

. said memory includes a default bit map table 
that includes a default bit nnap for each of said 
plurality of ports, each said default bit map in- 
cluding a bit corresponding to each of said 
bonded ports; and 40 
said port bonding circuitry programs each of 
said default bit maps to assign one bonded port 
of said bonded port set so as to achieve a rel- 
atively even distribution of bonded port assign- 
ments among said plurality of ports. 4S 

10. The network communication device of claim 8, said 
packets including broadcast packets that are re- 
ceived by any non-bonded ports to be sent to said 
bonded ports as determined by said port control cir- 50 
cuitry; wherein 

said port bonding circuitry monitors broadcast 
packets sent by each bonded port of said bonded 
port set and periodically re-programs each bit map 
of each of said plurality of entries so as to achieve 55 
a relatively even distribution of broadcast packet 
traffic among the bonded ports of said bonded port 
set. 



11. A packet-based network system, comprising; 

a first and a second network communication 
device according to any of claims 1 to 10; 
a bonded link coupled between the bonded port 
sets of said first and second network commu- 
nication devices, said bonded link including a 
connection link between respective bonded 
ports of said bonded port sets; 
said first network communication device select- 
ing any one of its bonded ports for each packet 
received by it to be sent to said second network 
communication device; and 
said second network communication device se- 
lecting any one of Its bonded ports for each 
packet received by it to be sent to said first net- 
work communication device. 

12. The network system of.claim 11, wherein said first 
and second network communication devices each 
select bonded ports to send packets so as to evenly 
distribute packets on said bonded port link. 

13. The network system of claim 12, wherein said first 
and second network communication devices select 
bonded ports on a packet by packet basts. 

14. The network system of claim 12. each packet in- 
cluding a unique source identifier, wherein said first 
and second network communication devices select 
bonded ports by assigning a bonded port to each 
source identifier. 

15. The network system of claim 14, wherein; 

first network communication device assigns 
source identifiers received by said first bonded 
port set to its bonded ports on an approximately 
equal basis; 

said first network communication device as- 
signs bonded ports among source identifiers 
received by its non-bonded ports on an approx- 
imately equal basis; 

said second network communication device as- 
signs source identifiers received by said sec- 
ond bonded port set to its bonded ports on an 
approximately equal basis; and 
said second network communication device as- 
signs bonded ports among source identifiers 
received by its non-bonded ports on an approx- 
imately equal basis. 

16. The network system of claim 11, wherein: 

said bonded port sets include bonded ports op- 
erating at different bandwidths; and 
wherein said first and second network commu- 
nication devices select bonded ports to distrib- 
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ute packets sent on each bonded port in pro- 
portion to its bandwidth. 

17. The network system of claim 11, further comprising: 

a third network communication device accord- 
ing to any of clairhs 1 to 1CJ; ' 
said first network communication device includ- 
ing a further plurality of ports and a further 
bonded port set Including at least two bonded 
ports; 

a second bonded link coupled between said 
third bonded port set of said third communica- 
tion device and said further bonded port set of 
said first network communication device, said 
bonded link including a connection link be- 
tween respective bonded ports of said third and 
further bonded port sets; 
for each packet received by said first commu- 
nication device to be sent to said third commu- 
nication device, said first communication de- 
vice selecting any one of its bonded ports; and 
for each packet received by said third commu- 
nication device to be sent to said first commu- 
nication device, said third communication de- 
vice selecting any one of its bonded ports. 

18. The network system of any of claims 11 to 17, 
wherein said first and second communication de- 
vices are network switches. 

19. A method of bonding ports of a network communi- 
cation device having a plurality of ports that send 
and receive packets, comprising steps of: 

defining at least two of the plurality of ports as 
bonded ports forming a bonded port set; 
receiving a packet; 

determining that the received packet is to be 
sent via the bonded port set; 
selecting one of the bonded ports; and 
sending the packet via the selected bonded 
port. 

20. The method of claim 1 9, further comprising a step 
of modifying a spanning tree function to prevent dis- 
abling any of the bonded ports. 

21 . The method of claim 1 9, wherein said selecting step 
comprises selecting bonded ports on a random ba- 
sis. 

22. The method of claim 1 9, further comprising steps of: 

retrieving a source identifier from the packet; 
determining a port identifier of the selected 
bonded port; 

storing the source identifier with the port iden- 



tifier of the selected bonded port; 
subsequently receiving a broadcast packet with 
an identical source identifier; 
retrieving the port identifier of the selected 
5 bonded port; and 

broadcasting the broadcast packet to selected 
ones of the plurality of ports including sending 
the broadcast packet via the selected bonded 
port. 

10 

23. The method of claim 22, further comprising steps of; 

receiving a plurality of packets at the non-bond- 
ed ports; 

'5 retneving a source identifier from each packet; 

for each unique source identifier, assigning any 
one of the bonded ports so as to evenly distrib- 
ute bonded port assignments among the 
unique source identifiers; and 

2^ storing each unique source identifier along with 

a port identifier of the corresponding assigned 
bonded port. 

24. The method of claim 19,furthercomprisingstepsof: 

2S 

receiving a packet at a bonded port; 
retrieving a source identifier from the packet; 
assigning any one of the bonded ports of the 
same bonded port set to the source identifier; 
50 and 

storing the source identifier along with a port 
identifier of the assigned bonded port. 

25. The method of claim 24, further comprising steps of: 

35 

receiving a plurality of packets at the bonded 
ports; 

retrieving a source identifier from each packet; 
for each unique source identifier, assigning any 

40 one of the bonded ports so as to evenly distrib- 

ute unique source identifiers among the bond- 
ed ports of the bonded port set; and 
storing each unique source identifier along with 
a port identifier of the corresponding assigned 

^ bonded port. 

26. A network system, comprising: 

a first network device including a plurality of 
50 ports; 

a second network device including a plurality of 
ports; 

a plurality of data links coupled between corre- 
sponding ports of said first and second network 
55 devices; and 

said first network device sending data to said 
second network device by distributing data 
among said at least two data links. 
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